Fundamental performance analysis for spoken dialogue systems based on a quantitative simulation approach
نویسندگان
چکیده
Currently the performance of dialogue system is mostly mearsured based on the analysis of a large dialogue corpus. In this way, the dialogue performance can not be obtained before the system is on line, and the dialogue corpus should be recollected if the system is modified. Also, the effect of different factors, including system’s dialogue strategy, recognition and understanding accuracy or user’s response pattern, etc., on the dialogue performance can not be quantitatively identified because they can not be precisely controlled in different corpora. In this paper, a fundamental performance analysis scheme for dialogue systems based on a quantitative simulation approach is proposed. With this scheme the fundamental performance of a dialogue system can be predicted and analyzed efficiently without having any real spoken dialogue system implemented or having any dialogue corpus actually collected. How the dialogue performance varies with respect to each factor, from recognition accuracy to dialogue strategy, can be individually identified, because all such factors can be precisely controlled in the simulation. The quality of service for the spoken dialogue system can also be flexibly defined and the design parameters easily determined. This approach is therefore very useful for the design, development and improvement of spoken dialogue systems, although the on-line system and real corpus will eventually be needed in the final evaluation and analysis of the system performance in any case.
منابع مشابه
Computer-aided Design/analysis for Chinese Spoken Dialogue Systems
Conventionally design principles for spoken dialogue systems are drawn either from experiences or from corpus-based analysis. However, human experiences are usually not precise enough for engineering design, while for corpus-based analysis many factors such as speech recognition or understanding performance and user’s behavior can never be precisely controlled. Recently, a new design/analysis a...
متن کاملExample-based training of dialogue planning incorporating user and situation models
To provide a high level of usability, spoken dialogue systems must generate cooperative responses for a wide variety of users and situations. We introduce a dialogue planning scheme incorporating user and situation models making such dialogue adaptation possible. Manually developing a set of dialogue rules to account for all possible model combinations, would be very difficult and obstruct syst...
متن کاملAnalysis of Emergency Department Queue System Performance: Simulation Approach Based on Experiment Design
Background: Simulation is an appropriate technique for analyzing and evaluating the dynamic behavior of complex systems. The present study aimed to develop an integrated model using a simulation approach based on designing experiments to analyze performance of the admission queue system of patients, who referred to the emergency department of the Modarres hospital. Methods: In this descriptive...
متن کاملReinforcement Learning-based Spoken Dialog Strategy Design for In-Vehicle Speaking Assistant
In this paper, the simulated annealing Q-learning (SA-Q) algorithm is adopted to automatically learn the optimal dialogue strategy of a spoken dialogue system. Several simulations and experiments considering different user behaviors and speech recognizer performance are conducted to verify the effectiveness of the SA-Q learning approach. Moreover, the automatically learned strategy is applied t...
متن کاملReward Shaping for Statistical Optimisation of Dialogue Management
This paper investigates the impact of reward shaping on a reinforcement learning-based spoken dialogue system’s learning. A diffuse reward function gives a reward after each transition between two dialogue states. A sparse function only gives a reward at the end of the dialogue. Reward shaping consists of learning a diffuse function without modifying the optimal policy compared to a sparse one....
متن کامل